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EXECUTIVE  SUH1ARY 


This  report  develops  a basis  for  understanding,  from  a management 
viewpoint,  the  process  of  accomplishing  product  assurance  and  test  functions 
during  Army  acquisition  programs.  To  achieve  this  purpose,  a management 
guide  for  Anry  project  managers  is  developed.  The  guide  emphasizes  what 
a manager  needs  to  know  and  do  about  product  assurance  and  test  as  he 
utilizes  this  functional  tool  to  help  him  achieve  his  program  goals.  Topics 
included  are  what  is  product  assurance  and  test,  how  the  function  fits  into 
the  acquisition  process,  what  work  has  to  be  done,  how  the  work  gets  done, 
and  management  considerations  for  planning  and  control. 
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SECTION  I 


INTRODUCTION 


In  today's  materiel  acquisition  environ- 
ment, the  project  manager 's*  challenge 
is  to  achieve  his  program  requirements  under  highly  structured  constraints. 
The  requirements  are  usually  set  in  the  areas  of  cost,  schedule,  and 
technical  performance.  The  constraints  usually  involve  staffing,  dollars, 
time,  organization,  authority  limits,  and  technical  risks.  In  this  environ- 
ment, the  project  manager's  task  is  to  integrate  the  diverse  program  elements 
toward  achievement  of  the  program  goals  while  striking  a program  balance  by 
a process  of  trade-offs  among  cost,  schedule,  and  technical  performance 
requirements  within  the  various  constraint  levels.  This  dynamic,  interactive 
situation  is  portrayed  in  Figure  1. 

The  obvious  question  to  be  posed  now  is: 
How  does  the  project  manager  cope  with 
this  rather  frightening  situation?  Besides  laving  nerves  of  steel,  the  pro- 
ject manager  has  a tool  set  at  his  disposal.  His  tools  are  the  experienced 
people  in  the  functional  areas  that  are  needed  to  accomplish  the  acquisition 
job.  These  functional  areas  are  business  management,  technical  management, 
configuration  management,  procurement  and  production,  integrated  logistic 
support,  and  product  assurance  and  test  (see  Figure  2).  The  description  of 
each  of  these  areas  and  the  interrelationships  between  them  throughout  the 


The  Project  Manager's 
Tool  Set 


The  Rroject  Manager's 
Challenge 


1 This  designation  also  includes  product  managers,  program  managers,  and 
commodity  managers. 
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Figure  1 
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Figure  2 


acquisition  process  is,  unfortunately,  a task  beyond  the  scope  of  this  paper. 
Because  of  the  concern  of  this  author  that  product  assurance  and  test  is  the 
least  understood  of  these  areas  among  project  managers,  this  paper  will 
focus  on  the  product  assurance  and  test  function. 

[Turpose  | The  purpose  of  this  paper  is  to  develop 

a basis  for  understanding,  from  a 

systems  engineering  viewpoint,  the  process  of  accomplishing  product  assur- 
ance and  test  functions  during  Army  acquisition  programs.  To  achieve  this 
purpose,  a management  guide  for  Army  project  managers  is  developed.  This 
guide  describes  the  product  assurance  and  test  work  breakdown  structure 
and  shows  how  this  structure  interfaces  with  the  overall  acquisition  process. 
| Intended  Use  1 The  guide  is  intended  to  be  used  by 

project  managers  to  obtain  an  overall 
perspective  of  the  management  aspects  of  the  product  assurance  and  test 
function.  It  emphasizes  what  a manager  needs  to  know  and  do  about  product 
assurance  and  test  as  he  utilizes  this  functional  tool  to  help  him  achieve 
his  program  goals.  In  this  regard,  the  guide  uses  a minimum  of  the  technical 
jargon  and  acronyms  of  the  practioners. 

The  remaining  sections  of  this  paper  are 
organized  in  the  following  fashion: 

• Section  II.  What  is  product  assurance  and  test  - the  philosophy 
and  need. 

• Section  III.  How  product  assurance  and  test  fits  into  the  acquisi- 
tion process  - the  integration  into  the  acquisition  framework. 

• Section  IV.  The  Work  Packages  - What  work  has  to  be  done. 


Organization  of 
the  Guide  
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• Section  V.  How  the  work  gets  done  - organizing,  staffing,  budgeting 
and  programming  aspects. 

• Section  VI.  Management  Considerations  - planning  and  control. 
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SECTION  II 

WHAT  IS  PRODUCT  ASSURANCE  AND  TEST? 


A Basis  for 
Definition 


Setting  aside  any  complicating  factors 
for  the  moment  and  relying  on  Webster's 
New  Collegiate  Dictionary,  one  can  begin  to  construct  a definition  of  the 
product  assurance  and  test  function  by  examining  the  implicit  meaning  of 
the  individual  words  in  the  phase.  A "product"  is  something  produced,  that 
is,  given  being,  form,  or  shape.  "Assurance"  is  the  act  of  assuring,  to 
give  confidence  to,  to  make  certain  the  attainment  of.  Test  is  to  "apply 
a critical  examination,  observation,  or  evaluation  as  a means  of  analysis 
or  diagnosis."  Based  on  these  meanings,  the  following  definition  is  posed: 

• .Product  Assurance  and  Test  Function  - a planned  and  systematic 

pattern  of  all  actions  necessary  to  provide  adequate  confidence  that 
the  product  will  perform  satisfactorily  in  service.2 


There  are  two  aspects  of  this  definition 
that  require  some  elaboration.  First, 
consider  the  phrase  "systematic  pattern  of  all  actions."  This  phrase  means 
that  the  product  assurance  and  test  function  can  be  viewed  as  a process 
which  is  applied  to  a system  throughout  the  system's  life  cycle.  This 
process  idea  is  a key  factor  in  understanding  the  function.  The  function  is 
not  a series  of  unrelated  tasks  performed  at  different  milestones  during 
materiel  acquisition.  Instead,  the  function  consists  of  a logical  sequence 
of  task  elements  that  begin  during  the  conceptual  phase.  Each  task  element 


Some 

Amplifications 


2 


™f  7nr?^i0uniS-ada?te^  from  that  for  "quality  assurance"  stated 
AMCP  706-100,  Design  Guidance  for  Producibil ity,"  August  1971 
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lays  the  ground  work  and  serves  as  a building  block  for  subsequent  task 
elements.  Thus,  the  function  must  be  applied  at  the  very  outset.  Its 
effectiveness  is  reduced  considerably  if  introduced  at  midstream  in  the 
acquisition  process. 

The  second  aspect  is  the  phrase  "to  provide  adequate  confidence."  The 
ideal  situation  in  acquisition  would  be  to  know  for  certain,  or  without  a 
doubt,  that  our  system  would  work  perfectly  when  it  was  deployed.  But, 
as  Hark  Twain  said,  only  two  things  in  life  are  certain  - death  and  taxes. 
With  apologies  to  Mr.  Twain,  another  certain  thing  is  that  acquisition 
programs  will  have  problems  that  were  not  anticipated.  The  project  manager 
lives  with  risk  and  uncertainty.  Basically,  his  entire  job  can  be  viewed 
as  one  of  defining  areas  of  risk  and  uncertainty  and  taking  actions  to 
deduce  these  areas  to  a tolerable  level.  Notice  the  emphasis  is  on  reduc- 
tion of  risk  and  uncertainty  - not  elimination.  Elimination  of  all  risk 
and  uncertainty  would  result  in  a state  of  certainty,  which  is  impractical 
if  not  impossible.  Here  is  where  the  notion  of  confidence  is  helpful.  The 
project  manager  needs  to  have  a sense,  or  feeling,  that  he  has  reduced  risk 
and  uncertainty  to  levels  where  he  can  continue  the  acquisition  program. 
Confidence  can  be  based  on  judgement  or  on  facts  and  figures.  The  former 
is  subjective  and  the  latter  is  objective.  Whether  subjective  or  objective, 
confidence  must  be  passed  on  to  higher  levels,  such  as  decision  bodies  like 
the  Army  and  Defense  Acquisition  Review  Councils.  Application  of  the  product 
assurance  and  test  function  can  help  identify  areas  of  risk  and  uncertainty 
and  assess  these  areas  for  confidence  that  risk  and  uncertainty  have  been 
reduced  to  acceptable  levels. 
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1 The  Need-]  Central  to  the  understanding  of  the 

function  itself  is  the  understanding 
of  the  need  for  the  function  in  the  acquisition  process.  From  a project 
management  viewpoint,  the  product  assurance  and  test  function  can  bridge 
the  gap  between  system  requirements  and  user  satisfaction,  as  shown  in 
Figure  3,  where  the  user  is  the  Army  troop  in  the  field.  This  perception 
is  based  on  the  premise  that  the  system  requirements  are  derived  from  the 
users'  needs.  In  this  context,  the  project  manager  needs  a degree  of 
confidence  that  his  acquisition  strategy  is  or  is  not  resulting  in  the 
procurement  of  a system  which  will  enjoy  user  satisfaction.  Product 
assurance  and  test  can  be  viewed  as  a management  control  tool  by  which  this 
confidence  is  achieved. 
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Figure 


SECTION  III 


HOW  PRODUCT  ASSURANCE  AND  TEST  FITS  INTO  THE  ACQUISITION  PROCESS 


Substantial  changes  in  the  acquisition 
process  were  directed  by  the  Office  of 
Management  and  Budget  (0MB)  in  October  1976  with  the  issuance  of  0MB 
Circular  A-109.3  This  circular  contains  the  following  management  objec- 
tives, among  others: 

• Ensure  that,  each  major  system  fulfills  a mission  need,  operates 
effectively  in  its  intended  environment,  and  demonstrates  a level 

of  performance  and  reliability  that  justifies  expenditure  of  limited 
resources. 

• Ensure  appropriate  trade-offs  among  investment  costs,  ownership 
costs,  schedules,  and  performance  characteristics. 

• Provide  strong  checks  and  balances  by  ensuring  adequate  system  test 
and  evaluation. 

• Maintain  a capability  to  assess  cost,  schedule  and  performance 
achievements  against  predictions  for  input  to  key  decision  points 
and  make  new  assessments  where  variances  occur.4 5 


DOD  implemented  the  changes  directed 
by  Circular  A-109  by  issuing  DOD 
Directive  5000.1  on  18  January  1977. 3 This  directive  basically  defines 
the  materiel  acquisition  process  for  major  systems  in  the  following  manner: 

0 Need  Statement  - Major  acquisition  programs  will  be  based  upon  an 
approved. Mission  Element  Need  Statement  (MENS).  The  MENS  will 
describe  the  need  against  a mission  and  this  need  will  be  stated 
in  operational  terms. 


the  Acquisition 
Process  Today 


0MB  Circular 
A-109 


3 0MB  Circular  A-109,  5 April  1976,  Major  System  Acquisitions. 

4 Ibid,  pp  4-5.  

5 DODD  5000.1,  18  January  1977,  Major  System  Acquisitions. 
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• Phases  An  acquisition  program  will  progress  through  four  sequential 
phases.  These  are  Conceptual,  Validation  and  Demonstration,  Full- 
Scale  Engineering  Development,  and  Production  and  Deployment. 

• Milestones  - There  are  four  key  milestones  in  the  process.  These 
are: 

• Milestone  0 - Approval  of  MENS. 

• Milestone  I - Approval  of  alternative  system  design  concepts. 

• Milestone  II  - Approval  of  recommended  alternative  for  system 
development. 

• Milestone  III  - Approval  of  production  and  deployment. 

• Decision  Reviews  - For  a major  system,  there  are  three  Defense  Acqui- 
sition Review  Councils  (DSARC).  These  are: 

• DSARC  I - Approval  to  enter  Validation  and  Demonstration  Phase. 

• DSARC  II  - Approval  to  enter  Full-Scale  Engineering  Development 


• DSARC  III  - Approval  to  enter  Production  and  Deployment  Phase. 

The  process  is  graphically  portrayed  in  Figure  4.  While  the  emphasis  is 
placed  on  major  system  acquisitions,6  DODD  5000.1  states  that  the  acquisi- 
tion policies  therein  should  be  used  as  guides  in  the  management  of  all 
acquisition  programs.  This  paper  will  focus  on  those  product  assurance  and 
test  tasks  required  for  major  system  acquisition;  however,  it  should  be 


recognized  that  the  task  principles  can  also  be  applied  to  non-major  system 
acquisitions.  The  tasks  will  be  the  same,  but  the  scope  of  effort  required 
will  be  less  for  non-major  system  acquisitions. 

The  product  assurance  and  test  function 
can  be  broken  down  into  definable  work 


Work  Breakdown 
Structure  Approach 


packages.  These  work  packages  have  unique  objectives  and  distinct  task 

6 Defined  as  a system  which  will  require  expenditure  of  over  $75  million  in 
Research,  Development,  Test  and  Evaluation  funds  or  $300  million  in  Pro- 
curement funds. 
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elements  associated  with  them.  Each  work  package  can  be  viewed  as  a sub- 
element of  the  overall  product  assurance  and  test  program,  which  is  a sub- 
element itself  of  the  system  acquisition  program.  This  approach  permits 
the  development  of  a work  breakdown  structure  for  the  function,  which  can 
be  integrated  into  the  work  breakdown  structure  for  the  system  acquisition 
program.  This  concept  is  shown  in  Figure  5.  The  integration  of  the 
functional  structure  with  the  hardware  structure  will  be  discussed  in 
Section  IV. 

A word  about  the  product  assurance  and 
test  objective  is  appropriate  before 
defining  the  work  packages.  The  objective  of  the  function  is  to  assure  user 
satisfaction.  This  objective  is  accomplished  by  the  following  activities: 

• Structuring  realistic  and  attainable  reliability,  availability,  and 
maintainability  (RAM)'  requirements. 

• Considering  RAM  characteristics  in  system  design. 

• Designing  controls  to  assure  system  meets  design  intent. 

• Utilizing  economical  and  realistic  test  designs  and  integrated 
testing. 

• Detailed  attention  to  quality  of  initial  and  follow-on  production, 
software,  technical  data,  repair  parts,  supply,  maintenance,  and 
foreign  sales  materiel. 

• Development  and  application  of  efficient  metrology  and  calibration 
procedures  and  equipment. 

• Assessment  of  system  status  and  problems  throughout  the  acquisition 
process. 

These  activities  are  used  as  the  basis  for  defining  the. work  packages. 


^ This  acronym  will  be  used  for  the  remainder  of  this  paper. 


Functional 

Objective 
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At  the  present  time,  the  product  assur- 
ance and  test  function  consists  of  four- 
teen work  packages.  These  packages  and  a brief  definition  of  each  are  listed 
below: 

• RAM  Requirements  - Those  activities  required  to  assure  that  RAM 
requirements  are  realistic,  attainable,  properly  stated,  and  capable 
of  test  and  assessment. 

• RAM  Engineering  - Those  engineering  activities  required  for  the 
proper  consideration  of  RAM  characteristics  in  the  design  process. 

• Quality  Engineering  - Activities  associated  with  the  development  of 
design  acceptance  provisions  in  Section  4 of  system  specifications 
and  the  design  of  special  acceptance  inspection  equipment. 

• Test  Management  - Those  activities  required  for  integrated  testing, 
including  test  design;  management  of  resources,  facilities  and  equip- 
ment for  tests;  test  evaluation;  and  test  program  management. 

• System  Assessment  (Development)  - Activities  related  to  the  assessment 
of  system  status  and  identification  of  problem  areas  during  the 
Conceptual,  Validation  and  Demonstration,  and  Full-Scale  Engineering 
Development  Phases. 

• Initial  and  Follow-on  Production  Quality  - Activities  intended  to 
assure  that  the  production  item  conforms  to  product  specifications. 

• Computer  Software  Quality  - Activities  related  to  achieving  an  accept- 
able level  of  quality  in  computer  software. 

• Technical  Data  Quality  - Those  activities  designed  to  assure  a high 
level  of  quality  in  technical  data  delivered  under  terms  of  contracts. 

o Repair  Parts  Quality  - Activities  to  assure  that  repair  parts  pro- 
cured for  system  support  exhibit  the  required  quality  levels  achieved 
in  initial  production. 

8 Supply  Quality  - Those  activities  for  assuring  that  major  and  second- 
ary replacement  items  exhibit  acceptable  quality  levels  when  received 
at  depots  and  when  shipped  from  depots. 

8 Maintenance  Quality  - Activities  which  assure  that  depot  maintenance 
does  not  compromise  materiel  quality. 
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• Foreign  Sales  Quality  - Activities  which  assure  that  materiel  pro- 
cured by  foreign  customers  conforms  to  quality  standards. 

• Metrology  and  Calibration  - Activities  associated  with  the  develop- 
ment  of  calibration  equipment  and  the  performance  of  wor Id-wide 
calibration  services. 

0 System  Assessment  (Deployment)  - Activities  for  the  assessment  of 
systems  in  field  use  to  identify  system  status  and  problem  areas  for 
corrective  actions. 

The  relationship  of  these  fourteen  work 
packages  to  the  acquisition  process  is 
depicted  in  Figure  6.  The  shaded  areas  of  this  matrix  indicate  the  phase 
during  which  the  individual  work  packages  must  be  performed.  Examination 
of  this  matrix  indicates  the  following  relationship  between  work  packages 
and  the  acquisition  phases: 

• Conceptual 

- RAM  Requirements 

- RAM  Engineering  (Planning  efforts) 

- System  Assessment  (Development) 

0 Validation  and  Demonstration 

- RAM  Engineering 

- Test  Management 

- System  Assessment  (Development) 

- Computer  Software  Quality 

- Metrology  and  Calibration 

0 Full-Scale  Engineering  Development 

- RAM  Engineering 

- Quality  Engineering 

- Test  Management 

- System  Assessment  (Development) 

- Initial  & Follow-on  Production  Quality  (Planning) 

- Computer  Software  Quality 

- Technical  Data  Quality 

- Metrology  and  Calibration 


Work  Packages  - Relationship 
to  Acquisition  Process 
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ACTIVITY  APPLICATION 


• Production  and  Deployment 

- Test  Management  (Production  Tests) 

- Initial  & Follow-on  Production  Quality 

- Computer  Software  Quality 

- Technical  Data  Quality 

- Repair  Parts  Quality 

- Supply  Quality 

- Maintenance  Quality 

- Foreign  Sales  Quality 

- Metrology  and  Calibration 

- System  Assessment  (Deployment) 

These  relationships  will  be  applicable  to  most  system  acquisition  pro- 
grams. They  can  be  used  as  a guideline  in  program  planning  for  the  four 
acquisition  phases.  In  Section  IV,  the  contents  of  the  individual  work 
packages  are  discussed. 
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SECTION  IV 


THE  WORK  PACKAGES 


I Purpose-]  The  product  assurance  and  test  function 

was  defined  in  Section  II.  In  Section 
III,  we  discussed  how  the  function  fits  within  the  acquisition  process  in 
terms  of  the  functional  work  packages.  In  this  section,  these  work  packages 
are  discussed  in  terms  of  what  task  elements  have  to  be  performed  for  each. 
The  intent  here  is  not  to  get  into  an  exhaustive  "how  to"  exercise,  but 
instead,  the  objective  is  to  provide  project  managers  with  an  appreciation 
for  the  type  and  scope  of  work  to  be  accomplished  under  these  work  packages. 

Before  describing  the  task  elements  of 
the  individual  work  packages,  an  under- 
standing of  how  the  hardware  and  functional  work  breakdown  structure  is 
integrated  will  be  helpful.  The  key  consideration  is  how  the  functional 
work  packages  are  applied  to  the  hardware  structure.  The  hardware  structure 
is  developed  by  a logical  process  of  breaking  down  the  system  into  work 
packages  that  define  unique  hardware  elements  for  which  work  can  be  easily 
applied  and  accounted.  These  hardware  elements  usually  satisfy  some  end 
use  function.  They  are  usually  those  specification  items  that  are  referenced 
directly  in  a contract  or  any  reparable  item  designated  for  separate  pro- 
curement. As  such,  these  hardware  elements  are  designated  as  configuration 
items  and  are  subject  to  configuration  management.  It  is  to  these  config- 
uration items  that  the  functional  work  packages  are  applied.  Thus,  the 
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functional  work  may  be  applicable  to  a major  sub-assembly,  component,  or 
repair  part;  depending  upon  the  degree  to  which  the  item  is  subjected  to 
configuration  control . 

1 RAM  Requirements  1 This  work  package  is  applied  at  the  out- 

set during  the  Conceptual  Phase.  Its 
application  is  usually  at  the  system  level  where  RAM  requirements  are 
generally  applicable.  The  objective  is  to  establish  cost  effective, 
realistic  and  feasible  requirements.  This  is  accomplished  by  validating 
the  quantification  of  requirements.  A key  task  element  is  the  development 
of  a RAM  baseline8  for  the  system  to  determine  what  are  the  feasible  RAM 
levels.  The  resulting  analysis  is  documented  in  a rationale  annex,  to  the 
requirements  document.  Failure  definitions  and  scoring  criteria  are  also 
developed  to  provide  the  groundrules  for  future  analyses  and  interpretation 
of  test  results.  These  activities  are  critical,  since  they  assure  that  the 
program  is  not  saddled  with  unrealistic,  unattainable  RAM  requirements  from 
the  beginning. 

I RAM  Engineering  ] Planning  for  this  work  package  begins 

during  the  Conceptual  Phase.  The 

engineering  effort  is  applied  during  the  design  activities  that  occur  during 
the  Validation  and  Demonstration  and  Full-Scale  Engineering  Development 
Phases.  The  objective  is  to  achieve  RAM  requirements  during  development. 

The  task  elements  for  this  work  package  are: 


8 RAM  baseline  is  a quantitative  assessment,  based  on  data  from  similar 
systems  and  engineering  judgements,  of  the  state  of  the  art  for  RAM, 
which  can  be  used  in  evaluating  RAM  requirements  for  the  new  system. 
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• RAH  Growth  Planning  and  Management  - This  task  involves  establishing 
planned  values  for  RAM  at  successive  points  in  the  system  acquisition, 
assessment  of  RAM  achievements  at  these  points,  predictions  of  RAM 
growth  trends,  and  the  identification  of  time  and  resources  needed  to 
achieve  the  required  RAM  levels. 

• RAM  Input  to  Coordinated  Test  Plans  - This  input  consists  of  RAM 
requirements,  failure  definitions  and  scoring  criteria,  sample  size, 
test  design,  testing  risks,  environmental  profiles,  operational  mode 
summary  and  mission  profiles  and  the  data  collection  scheme.  Test 
data  are  analyzed  and  results  evaluated. 

• RAM  in  Contracts  - This  task  element  calls  for  appropriate  RAM  con- 
siderations to  be  inserted  into  significant  development  and  produc- 
tion solicitations  and  resultant  contracts.  The  considerations 
involve  requirements  for  hardware  RAM,  RAM  test  and  demonstration, 

RAM  programs,  RAM  management  controls,  and  incentives  and  warranties. 

• RAM  Design  Evaluation  and  Analysis  - Application  of  design  practices 
for  RAM  achievement  are  the  crux  of  this  element.  These  practices 
include  parts  qualification,  stress  analysis,  part  derating,  worse 
case  analysis,  failure  modes  and  effects  analysis,  component  control, 
in-depth  design  reviews,  predictions  and  allocations,  and  failure 
analysis. 


Quality  engineering  activity  takes 
place  during  the  Full-Scale  Engineering 
Development  Phase.  The  objective  is  to  establish  controls  to  assure  the 
system  meets  the  design  intent.  Quality  engineering  task  elements  are  appli- 
cable to  the  total  system,  major  assemblies,  repair  parts,  and  critical 
processes.  These  tasks  develop  criteria  for  assuring  conformance  with 
design  intent  during  production,  storage,  and  depot  maintenance  operations. 
The  task  elements  include  the  development  of  quality  standards,  preparation 
of  quality  assurance  provisions  of  specifications,  design  of  inspection 
equipment,  test  design  for  configuration  item  qualification  and  control, 
development  of  serviceability  standards  for  materiel  in  storage,  preparation 
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of  quality  assurance  provisions  for  rebuild/overhaul  specifications,  and 
development  and  application  of  nondestructive  testing  techniques. 

Test  Management  activity  begins  in  the 
Validation  and  Demonstration  Phase  and 
ends  in  the  Production  and  Deployment  Phase.  The  objective  is  to  execute 
an  integrated  test  program  that  provides  sufficient  testing  consistent  with 
the  acquisition  objectives  and  time  and  resource  constraints.  Tests  involve 
those  that  influence  design,  such  as  contractors'  engineering  development 
testing,  and  those  that  assess  design,  such  as  Government  development/ 
operational  tests.  In-house  laboratory  testing  for  component  qualification, 
first  article,  and  failure  analysis  is  also  included  in  this  work  package. 

The  emphasis  is  on  the  integration  of  contractor  and  Government  testing  as 
well  as  on  integration  of  development  and  operational  testing  consistent 
with  the  objectives  of  each  type  of  testing.  The  overall  purpose  of  testing 
is  to  reduce  risk.  This  purpose  generally  drives  the  amount  of  testing  that 
has  to  be  done.  However,  when  unknowns  arise  that  were  not  anticipated, 
additional  testing  beyond  that  programed  may  have  to  be  done.  This  situa- 
tion usually  requires  the  prudent  manager  to  hedge  by  holding  a management 
reserve  as  a contingency.  The  key  task  elements  for  test  management 
include  the  following: 

• Test  Objectives  - Test  objectives  are  developed  for  the  system 
acquisition  program  to  guide  test  planning. 

• Test  and  Evaluation  Master  Plan  (TEMP)  - The  TEMP  is  the  master  plan 
which  documents  all  requirements,  tasks,  resources,  schedules,  mile- 
stones, facilities,  equipment,  and  coordination  points  for  the 
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acquisition  testing  and  evaluation  program.  It  serves  as  the  basis 
for  programming  and  budgeting  decisions  related  to  testing.  Also, 
it  is  used  as  a control  document  to  monitor  commitment  of  resources 
and  test  progress.  The  preparation  of  the  TEMP  is  initiated  in  the 
Validation  and  Demonstration  Phase  and  is  updated  as  new  information 
and  planning  estimates  become  available.  The  importance  of  the  TEMP 
in  achieving  an  effective  and  economical  test  program  should  be 
recognized  by  project  managers. 

• Test  Resources  - Resources  required  for  testing,  such  as  people, 
facilities,  and  equipment,  are  identified  and  actions  taken  to 
make  them  available  when  they  are  needed. 

« Testing  Focal  Point  - A Test  Integration  Working  Group  is  formed. 
This  group  is  chaired  by  the  project  manager's  representative  and 
includes  representatives  from  cognizant  agencies,  such  as  TRADOC, 
OTEA,  TECOM,  and  AMSAA.  The  group  accomplishes  test  planning, 
design,  and  data  scoring. 


The  objective  of  system  assessment  in 
development  is  the  quantification  and 
analysis  of  system  performance  in  order  to  track  achievement  of  requirements, 
predict  future  results  and  anticipate  shortfalls,  identify  problems,  and 
verify  corrective  actions.  This  task  element  can  be  applied  to  the  total 
system  or  any  of  its  configuration  items.  The  identification  of  system 
parameters  for  assessment  and  initial  planning  occur  in  the  Conceptual 
Phase,  while  detailed  assessments  are  performed  during  the  Validation  and 
Demonstration  and  Full-Scale  Engineering  Development  Phases.  The  task 
elements  are  as  follows: 

9 System  Assessment  Status  Reports  - These  reports  document  the  results 
of  periodic  assessments  of  key  system  characteristics,  such  as  relia- 
bility, availability,  maintainability,  and  system  effectiveness. 

These  key  characteristics  are  typically  contained  in  Decision 
Coordinating  Papers,  Army  Program  Memorandums,  Selected  Acquisition 
Reports,  Department  of  Army  Program  Reviews,  and  Review  and  Command 
Assessment  of  Projects  (RECAP)  Briefings.  The  system  assessments 
permit  the  quantification  of  these  characteristics  and  assure  con- 
sistent reporting  to  higher  authority. 
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« Data  Base  - A key  element  of  any  acquisition  program  is  the  mass  of 
data  on  system  performance  generated  in  contractor  and  Government 
testing.  It  is  crucial  to  program  management  that  this  data  be 
carefully  organized  and  controlled  due  to  its  importance  in  assessing 
program  progress  and  measuring  the  achievement  of  requirements. 

This  task  emphasizes  the  orderly  collection,  storage,  retrieval 
and  analysis  of  program  data  to  support  status  assessment,  problem 
identification  and  management  decision-making. 

• Work  Package  Assessment  - This  task  involves  the  detailed  technical 
and  engineering  assessment  of  contract  work  packages  in  those  cases 
where  the  contractor's  actual  cost  has  significantly  exceeded  his 
budgeted  cost  for  the  work  package.  The  purpose  of  this  type  of 
assessment  is  the  early  identification  of  technical  problems  and 
evaluation  of  their  impact  on  program  costs  and  schedule.  Ideally, 
it  is  performed  by  a Government  team,  but  may  be  included  as  a 
provision  of  the  contractor's  scope  of  work.  In  the  latter  case, 
the  contractor  should  be  required  to  submit  a report  on  findings. 

• Quality  Readiness  Review  - This  assessment  occurs  prior  to  the 
decision  to  enter  production.  Its  purpose  is  to  assure  that  the 
product,  material , and  process  specifications  for  the  configuration 
items  properly  characterize  the  configuration  item  for  production. 
Emphasize  is  placed  on  the  ability  to  achieve  consistency  in  pro- 
duction through  the  proper  specification  of  product  attributes, 
control  of  appropriate  materials  and  processes,  and  the  utiliza- 
tion of  efficient  and  effective  tests  and  inspections  for  deter- 
mining product  attribute  compliance  with  specification  requirements. 
This  assessment  should  be  performed  by  a Government  team. 

Initial  efforts  on  this  work  package 

begin  in  the  Full-Scale  Engineering 


Development  Phase.  The  objectives  are  to  apply  cost  effective  quality 
assurance  contract  provisions,  provide  confidence  in  product  acceptability, 
perform  first  article  tests,  and  assure  proper  releases  of  materiel  to  the 
field  users.  Since  this  work  package- is  an  integral  part  of  the  production 
process,  its  task  elements  are  closely  tied  to  the  process  of  contract 
award  and  execution.  They  can  be  considered  in  terms  of  pre-solicitation, 
pre-award  and  post-award  activities: 
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• Contract  Requirements  - The  output  of  this  task  provides  input  to 
the  request  for  proposal  in  terms  of  a contemplated  structure  of 
quality  assurance  requirements  for  the  anticipated  contract.  These 
requirements  are  generally  based  on  the  analysis  of  the  technical 
data  package  and  participation  in  the  physical  configuration  audit. 

• Pre-Award  Survey  - This  survey  is  usually  performed  by  the  cognizant 
contract  administration  service  component,  either  Army,  Navy,  Air 
Force,  or  Defense  Contract  Administration  Service.  Its  purpose 

is  to  evaluate  the  capability  of  a prospective  contractor  to  comply 
with  the  terms  of  the  proposed  contract.  The  particulars  for  the 
survey  are  set  forth  in  Section  1-905.4  and  Appendix  K of  the  Armed 
Services  Procurement  Regulation  (ASPR).  An  appraisal  of  the  con- 
tractor's quality  control  system  is  a key  element  in  the  survey. 

The  project  management  office  should  receive  the  pre-award  survey 
findings  for  the  contractor  who  is  awarded  the  production  contract. 

• Participation  in  Production  Readiness  Reviews  - DOD  Directive  5000.34, 
Defense  Production  Management,  establishes  the  policy  that  production 
decisions  shall  be  supported  by  an  assessment  of  the  program  readiness 
for  production.  This  assessment  is  to  be  based  on  a formal  production 
readiness  review.  The  purpose  of  this  review  is  to  determine  whether 
the  system  under  development  is  ready  for  efficient  and  economical 
quantity  production,  that  all  important  production  engineering  pro- 
blems encountered  during  development  have  been  resolved,  and  that 

the  contractor  has  accomplished  planning  for  the  production  phase. 
Since  quality  considerations  are  deeply  imbedded  in  the  production 
process,  it  is  critical  that  personnel  with  quality  expertise  parti- 
cipate in  the  production  readiness  review,  and  that  the  readiness  of 
the  contractor's  quality  system  for  quantity  production  be  thoroughly 
assessed. 

• Initial  Production  - The  quality  program  provisions  of  the  contract 
are  implemented  by  the  contractor  during  initial  production.  The 
responsibility  for  monitoring  the  contractor's  implementation  and 
performance  of  the  production  quality  program  belongs  to  the  cogni- 
zant contract  administration  service  component.  The  project  manage- 
ment office  is  responsible  for  issuing  any  special  instructions  to 
the  contract  administration  service  component  regarding  the  admini- 
stration of  the  quality  provisions.  This  is  accomplished  by  the 
Quality  Assurance  Letter  of  Instruction.  Although  monitoring  quality 
is  a responsibility  of  the  cognizant  contract  administration  service 
component,  the  project  or  functional  support  personnel  may  perform 
periodic  key  inspections.  These  surveys  are  restricted  to  an  inspec- 
tion of  the  product  on  the  production  line.  Product  deficiencies 
discovered,  however,  may  dictate  an  overall  review  of  the  contractor's 


9 DOD  Directive  5000.34,  Defense  Production  Management,  signed  coordination 
draft,  dated  27  September  1977.  As  of  27  October  1977,  the  Directive  had 
not  been  published. 
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quality  program  in  conjunction  with  the  cognizant  contract  administra- 
tion service  component.  A post-award  conference  may  also  be  performed 
by  the  cognizant  contract  administration  service  component  during 
initial  production.  The  objectives  of  this  conference  are  to  make 
sure  the  contractor  understands  the  contract  and  to  inform  the  con- 
tractor of  any  problems  experienced  in  previous  production  of  the 
system. 

• First  Article  - A key  event  in  the  initial  production  period  is  the 
delivery  of  the  first  article  produced  for  Government  tests  and 
inspections.  The  conformance  of  the  first  article  is  a crucial  test 
of  the  capability  of  the  contractor's  production  process  to  deliver 
a product  which  meets  specification  requirements.  The  tests  are 
performed  both  in-plant  and  at  Government  test  grounds.  Due  to  their 
critical  nature,  it  is  advisable  to  have  project  or  functional  support 
personnel  witness  these  tests. 

• Materiel  Releases  - Once  the  first  article  tests  have  been  success- 
fully  completed,  subsequent  production  can  be  delivered  to  inventory 
points.  This  equipment  must  be  certified  by  the  project  manager  or 
functional  commander  for  release  to  field  users.  The  objective  is  to 
assure  that  the  user  will  receive  equipment  that  performs  properly 
and  can  be  supported.  The  factors  that  must  be  considered  in  a 
materiel  release  decision  include  performance,  repair  parts,  basic 
issue  items,  manuals,  maintenance  equipment,  and  training. 

• Comparison  Tests  - During  volume  production,  periodic  comparison 
tests  should  be  performed  to  assure  that  the  production  items  are 
compliant  with  specification  requirements.  These  tests  are  compar- 
able in  scope  to  the  first  article  tests. 

The  major  activities  of  this  work  package 

occur  during  the  Validation  and  Demon- 


stration, Full-Scale  Engineering  Development,  arid  Production  and  Deployment 
phases.  The  objective  is  to  assure  the  development  and  procurement  of  com- 
puter software  that  is  reliable  and  operationally  effective.  Due  to  its 
special  differences  from  hardware,  careful  attention  has  to  be  paid  to 
computer  software  development  and  procurement.  The  most  critical  process  is 
the  design  and  testing  bf  the  software.  Task  elements  of  this  work  package 
are  as  follows: 
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• Requirements  Quantification  - Software  design  is  more  of  an  art  than 
a science.  No  two  programmers  are  likely  to  approach  a design  in 
the  same  manner.  Thus,  it  is  essential  that  some  measure  of  software 
reliability  be  defined  as  a design  requirement.  The  definition  of 

a reliability  requirement  should  occur  in  the  Validation  and  Demon- 
stration  phase  where  initial  software  design  approaches  are  being 
developed.  Care  should  be  taken  to  assure  that  the  requirement  is 
realistic  and  relevant  to  the  software's  functions.  It  should  also 
be  stated  in  a manner  that  permits  its  measurement  during  testing. 

• Software  in  Contracts  - This  task  element  applies  to  contracts  in  the 
Full-Scale  Engineering  Development  phase.  In  these  contracts,  soft- 
ware aspects  must  be  defined  in  the  scope  of  work,  system  specifi- 
cation, and  data  requirements.  The  scope  of  work  should  specify  all 
the  software-related  tasks  to  be  performed  under  the  contract.  Soft- 
ware requirements  and  compliance  criteria  is  spelled  out  in  the 
system  specification.  Documentation  of  the  software  design  logic 
should  be  required  as  a deliverable  item.  This  documentation  is 
essential  for  controlling  software  configuration. 

• Software  Evaluation  and  Analysis  - This  task  is  a never-ending  one 
for  software.  During  development,  design  reviews  are  the  principal 
means  of  evaluating  the  progress  of  software  design  efforts.  Once 
formal  configuration  control  is  invoked,  each  software  change  must 
be  carefully  evaluated  and  tested  to  assure  it  does  not  adversely 
impact  the  total  program. 

• Testing  - Software  testing  is  critical  in  proving  that  the  software 
will  reliably  perform  its  intended  function.  The  testing  should  be 
performed  in  an  operationally  realistic  environment  with  representative 
hardware.  The  testing  procedures  and  ground  rules  for  assessing  test 
results  must  be  documented  and  agreed  to  by  all  cognizant  agencies. 

The  software  design  should  be  complete  before  demonstration  type 
testing  is  conducted. 


This  work  package  is  performed  during 
the  Full-Scale  Engineering  Development 
and  Production  and  Deployment  phases.  The  objective  is  to  obtain  a high 
quality  technical  data  package  for  use  in  competitive  procurements  of  future 
buys  of  production  systems  and  secondary  items  for  supply  support.  The  task 
elements  include  structuring  quality  provisions  for  technical  data  as  inputs 

i 

to  solicitations  and  contracts,  evaluation  and  control  of  technical  data, 
and  verifying  technical  data  accuracy  during  the  physical  configuration  audit. 

i 
} 
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The  objective  of  this  work  package,  which 
is  performed  during  the  Production  and 
Deployment  phase,  is  to  assure  that  reprocurements  of  repair  parts  exhibit 
comparable  quality  as  the  initial  production  items.  To  accomplish  this, 
technical  data  for  repair  parts  must  contain  quality  requirements  and 
acceptance  provisions  which  will  assure  the  delivery  of  quality  materiel. 
Excessive  use  of  waivers  and  deviations  to  facilitate  delivery  must  be 
avoided.  Acceptance  tests  should  be  designed  to  assure  repair  part  compati- 
bility with  overall  system  requirements. 

I Supply  Quality  | The  quality  of  supply  items  furnished 

field  users  is  a subject  deserving  con- 
tinuing concern.  Performed  during  supply  operations  in  the  Production  and 
Deployment  phase,  the  objective  of  this  work  package  is  to  assure  that 
supplies  entering  the  inventory  and  being  issued  to  troops  possess  an 
acceptable  level  of  quality.  The  task  elements  of  this  work  package  are 
usually  performed  at  the  depot  which  receives  and  issues  supplies.  The 
first  task  element  is  the  establishment  and  operation  of  a system  for 
inspecting  incoming  supplies  from  production  sources.  The  second  element 
is  the  establishment  and  operation  of  a system  for  inspecting  supplies  as 
they  are  issued  to  field  users.  Both  these  systems  must  utilize  lot  sampling 
and  inspection  criteria  which  reflects  production  specifications. 

This  work  package  is  performed  during 
the  Production  and  Deployment  phase 
and  is  applicable  to  maintenance  activities  performed  at  the  depot  level 
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and  field  maintenance  levels  performing  extensive  repair/rebuild  work.  The 
objective  is  to  assure  that  such  maintenance  does  not  degrade  equipment 
quality.  The  task  elements  include  development  of  quality  standards  for 
inspection  and  acceptance  of  rebuilt  and  overhauled  equipment,  performance 
of  in-process  inspection,'  conducting  acceptance  inspections  and  tests,  and 
analyzing  and  evaluating  the  quality  of  maintenance. 

This  work  package  only  applies  to  items 
of  equipment  procured  for  foreign  cus- 
tomers. Its  objective  is  to  assure  that  the  equipment  delivered  to  foreign 
customers  exhibits  acceptable  quality.  The  initial  task  element  is  the 
audit  of  documents  for  Price  and  Availability  Requests  and  Letters  of  Offer 
and  Acceptance  to  assure  accuracy  and  completeness  of  the  required  entries. 
Once  the  contracting  process  is  started,  appropriate  quality  provisions  are 
included  in  solicitations  and  contracts  consistent  with  the  foreign  customer's 
desires.  Prior  to  shipment,  inspections  are  performed  to  assure  materiel 
compliance.  After  delivery  to  the  foreign  customer,  quality  assurance  teams 
perform  in-country  visits  to  resolve  any  complaints  received  from  the 
customer  when  he  takes  delivery. 

Metrology  and  calibration  tasks  are  per- 
formed during  the  acquisition  phases  of 
Validation  and  Demonstration,  Full-Scale  Engineering  Development,  and  Produc- 
tion and  Deployment.  The  objective  of  this  work  package  is  to  assure  that 
adequate  metrology  and  calibration  capabilities  are  developed  during  the 
acquisition  process.  This  is  accomplished  by  identifying  metrology  and 
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calibration  needs  during  the  design  activity.  Alternatives  are  developed 
for  satisfying  tnese  needs  by  using  either  system-dedicated  calibration 
equipment  or  through  utilization  of  multi-purpose  equipment.  A decision 
must  be  made  early  in  the  program  whether  to  develop  the  needed  equipment 
or  procure  existing  commercial  equipment  which  will  satisfy  the  needs.  The 
equipment  is  then  procured  or  developed  and  tested  in  conjunction  with  the 
hardware  testing.  Provisions  must  also  be  made  for  the  training  and  deploy- 
ment of  personnel  specialists  to  perform  the  calibration  functions.  At  this 
point,  consideration  should  be  given  to  the  utilization  of  facilities  of  the 
other  services.  Once  the  system  is  deployed,  a program  for  performing  the 
required  calibration  cycles  has  to  be  developed  and  implemented. 

The  final  work  package  to  be  discussed 
is  that  for  system  assessment  of  deployed 
equipment.  A key  aspect  of  the  product  assurance  and  test  philosophy  of 
assuring  user  satisfaction  is  that  of  knowing  how  the  equipment  is  per- 
forming in  the  field  and  what  problems  the  field  user  has  with  it.  This 
activity  is  an  essential  prerequisite  to  developing  and  implementing 
effective  corrective  actions.  Emphasis  is  placed  on  the  assessment  of 
the  total  system,  which  includes  hardware  performance,  logistic  support, 
maintenance  support,  training,  technical  manuals,  operator  performance, 
operational  readiness,  safety,  and  technical  support.  The  task  elements 
for  this  work  package  include  preparation  of  system  assessment  letters, 
conduct  of  disciplined  reviews,  and  implementation  of  RAM  improvements  on 
selected  equipment. 
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• System  Assessment  Letters  - This  task  element  requires  that  the 
functional  commanders  conduct  annual  assessments  of  those  deployed 
systems  which  are  crucial  to  the  soldier's  ability  to  move,  shoot, 
and  conmunicate.  Headquarters,  DARCOM  establishes  an  annual 
schedule  for  accomplishing  these  assessments.  The  functional 
commanders  document  their  findings  in  a formal  report.  Tin's  report 
is  forwarded  to  Headquarters,  DARCOM  under  a personal  letter  from 
the  functional  commander  which  contains  his  overall  personal  assess- 
ment of  the  system's  status  and  problems.  The  assessment  report 
contains  a system  improvement  plan  for  correcting  any  problem  areas 
surfaced  by  the  assessment.  Headquarters,  DARCOM  staff  reviews 
the  assessment  report  and  the  functional  commander's  letter  is 
responded  to  by  the  Deputy  Commander  for  Materiel  Readiness,  DARCOM. 
This  process  assures  management  visibility  of  the  status  of  equip- 
ment in  the  field. 

0 Disciplined  Reviews  - The  disciplined  review  is  a joint  assessment 
of  a fielded  system  by  the  operational  user  conmand,  DARCOM  func- 
tional colander,  and  the  Training  and  Doctrine  Command  (TRADOC). 

The  DARCOM  functional  command  schedules  and  administers  the  review. 
Typically,  the  reviews  will  be  held  at  two  points  in  the  operational 
life  of  a system,  after  initial  deployment  (2-3  years)  and  at  the 
mid-point  of  the  systems  estimated  useful  life.  The  review  is 
preceded  by  each  of  the  participants  conducting  their  own  assess- 
ments of  the  system.  The  review  is  a formal,  face-to-face  meeting 
with  a structured  agenda  which  features  presentations  of  all  parti- 
cipants points  of  view.  It  culminates  with  a jointly  approved 
system  improvement  plan.  This  type  of  review  is  intended  to  improve 
corminications  with  the  field  user  and  trainer  on  problems  and 
improvement  actions  for  operational  systems. 

t RAM  Improvement  of  Selected  Equipment  - The  payoff  of  system  assess- 
ments and  disciplined  reviews  is  the  identification  of  initiatives 
to  improve  the  readiness  and  reduce  operating  and  support  costs  for 
operational  systems.  Known  by  the  acronym  RISE,  this  task  element 
seeks  to  utilize  a systematic  approach  for  improving  the  RAM  of 
fielded  equipment.  A typical  RISE  program  consists  of  four  phases 
of  effort.  First,  candidates  for  improvement  are  identified  through 
assessments.  Second,  these  candidates  are  analyzed  from  design  and 
cost  viewpoints  zo  determine  feasibility  and  estimated  cost  savings. 
Third,  the  candidates  with  the  greatest  potential  payoffs  are  imple- 
mented through  product  improvements  or  engineering  change  proposals. 
Lastly,  the  improvements  are  assessed  after  deployment  to  verify 
the  degree  of  readiness  improvement  or  cost  savings  actually  achieved. 
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SECTION  V 


HOW  THE  WORK  GETS  DONE 


I Purpose!  Now  that  the  reader  has  an  apprecia- 

tion for  the  fourteen  product  assurance 
and  test  functional  work  packages  and  their  task  elements,  it  is  appropri- 
ate to  discuss  in  this  section  how  the  work  gets  done.  The  topics  to  be 
covered  include  the  organizing,  staffing,  and  budgeting  and  programming 
associated  with  the  product  assurance  and  test  function. 

Before  getting  to  the  specifics  on  the 
topics  mentioned  above,  the  question 
of  how  much  work  should  be  done  needs  to  be  discussed.  The  answer  that 
immediately  comes  to  one's  mind  is  that  the  amount  of  product  assurance 
and  test  work  that  needs  to  be  done  will  depend  upon  the  situation  in  which 
the  system  acquisition  can  be  expected  to  take  place.  In  assessing  what 
this  situation  might  be,  the  life  cycle  characteristics  spectrum  should  be 
defined  and  related  to  the  product  assurance  and  test  function.  A typical 
spectrum  might  be  characterized  by  elements  of  uncertainty,  risk,  alter- 
natives, cost,  requirements,  planning,  trade-offs,  schedules,  and  changes. 
Each  of  these  elements  can  be  viewed  as  representing  a continuum  of  possi- 
bilities and  the  situational  aspects  of  the  acquisition  can  be  located  in 
a relative  position  on  each  continuum  to  develop  the  overall  life  cycle 
characteristics  spectrum.  ..In  a similar  fashion,  a continuum  for  product 
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assurance  and  test  work  can  be  developed  and  related  to  the  continuum  for 
each  element.  This  approach  is  illustrated  in  Figure  7.  For  example, 
more  product  assurance  and  test  work  will  have  to  be  done  if  there  is 
uncertainty,  high  risk,  many  alternatives,  flexible  schedules,  vague 
requirements,  flexible  planning,  open  trade-offs,  and  many  changes  than 
if  the  converse  were  true.  Assessing  the  acquisition  program  in  this 
fashion  will  aid  the  project  manager  in  determining  how  much  of  his 
resources  should  be  dedicated  to  the  product  assurance  and  test  function. 

I Organizing  I There  is  no  single  answer  to  the 

question  of  how  to  organize  to  accom- 
plish the  product  assurance  and  test  work.  Rather,  the  answer  lies  in  an 
assessment  of  the  life  cycle  characteristics  spectrum  and  making  a judge- 
ment on  the  best  approach  based  on  the  facts  of  the  situation.  Generally 
speaking,,  the  more  v/ork  that  has  to  be  done,  the  more  structured  the 
organization  must  be.  This  is  an  inevitable  consequence  of  having  to 
dedicate  substantial  resources  to  the  function--resources  which  must  be 
carefully  managed.  Thus,  the  organization  should  be  designed  to  facilitate 
degree  of  management  required.  There  are,  however,  some  approaches  which 
can  be  used  to  guide  decision-making  on  what  organizational  structure  to 
use. 

• Project  Dedicated  - This  structure  is  depicted  in  Figure  8.  In  this 
approach,  all  the  resources  needed  to  perform  the  work  are  located 
within  the  project  itself.  The  project  is  autonomous  and  does  not 
have  to  rely  on  the  functional  command  for  support.  This  approach 
is  costly  and  should  only  be  considered  for  large,  complex  projects. 
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• Project/Functional  Mix  - Depicted  in  Figure  9,  this  approach  features 
a core  of  functional  specialists  for  management  and  integration  of 
the  product  assurance  and  test  effort  located  in  the  project  office. 
The  predominance  of  the  work  is  performed  by  the  functional  support 
organization  of  the  functional  command.  This  approach  results  in 
less  functional  duplication  but  requires  a greater  degree  of  inte- 
gration on  the  part  of  the  project  personnel. 

• Project  Integrator  - This  approach,  depicted  in  Figure  10,  places  a 
very  small  core  in  the  project  office.  Their  task  is  limited  to 
integration  of  the  product  assurance  and  test  work  with  the  other 
functional  disciplines.  The  management  and  performance  of  the  work 
must  be  performed  by  the  functional  support  organization  of  the 
functional  command.  Under  this  arrangement,  the  project  manager 
has  less  control  over  the  performance  of  the  work. 

Again,  these  are  alternative  approaches--the  best  approach  will  depend  upon 

the  situation.  As  resources  become  harder  to  obtain,  however,  the  trend 

will  probably  be  more  toward  the  project/functional  mix  and  the  project 

integrator  approaches. 

| Staffing  | Regardless  of  the  type  of  organizational 

approach  choosen,  the  satisfactory 

accomplishment  of  the  product  assurance  and  test  work  will  depend  on  staffing 
that  organization  with  capable  people.  A few  words  about  the  type  of  skills 
needed  to  perform  the  work  might  be  helpful  in  staffing  decisions.  First, 
one  must  realize  that  there  is  not  a separate  career  field  for  civilians 
nor  an  occupational  specialty  for  military  personnel  in  the  field  of 
product  assurance  and  test.  If  an  examination  of  current  product  assurance 
and  test  organizations  were  made,  one  would  find  a mix  of  engineers  and 
scientists,  mathematicians  and  statisticians,  qual ity  assurance  specialists, 
and  various  technicians.  These  personnel  have  been  educated  in  their 
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primary  specialty  and  usually  have  some  “hands-on11  work  experience  in  that 
specialty.  Some,  however,  have  entered  the  product  assurance  and  test  work 
directly  from  their  educational  experiences.  Product  assurance  and  test 
expertise  is  usually  obtained  from  formal  education  in  allied  disciplines, 
on-the-job  training,  or  formal  Government  training  in  the  prcouct  assurance 
and  test  discipline.  In  selecting  a staff,  the  key  consideration  should  be 
an  assessment  of  an  individuals  potential  to  perform  the  work.  This 
assessment  should  consider  the  individuals  work  experience  and  educational 
background.  Where  practicable,  an  interview  is  a good  method  for  assessing  . 
the  individual's  inclination  and  motivation  for  performing  the  product 
assurance  and  test  work.  Generally  speaking,  individuals  strong  in  fact- 
finding ability  and  quantitative  analysis  can  perform  the  work  well. 
Experience  with  the  hardware  being  developed  is  not  a necessary  prerequisite 
for  performing  the  functional  work,  but  such  experience  may  provide  a com- 
petitive advantage  to  the  individual  who  possesses  it. 

It  almost  goes  without  saying  that  if 
funds  are  not  made  available,  none  of 
the  work  will  be  accomplished.  Careful  programming  and  budgeting  can  assure 
that  sufficient  funds  will  be  available  at  the  time  they  are  needed.  There 
is  not,  however,  a separate  budget  line  for  funding  product  assurance  and 
test  work  in  the  current  Army  financial  structure.  The  work  may  be  described 
as  the  level  of  effort  type,  with  this  effort  being  directly  applied  on  a 
specific  system  acquisition.  This  is  particularly  true  during  the  develop- 
ment and  production  effort.  Once  an  item  is  out  of  production,  however. 
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the  work  activities  related  to  deployed  systems  is  usually  applied  across 
the  board  for  all  systems  and  is  funded  from  the  O&MA  account.  The  fact 
that  the  work  is  level  of  effort  oriented  requires  that  detailed  attention 
be  paid  to  defining  the  level  of  effort  and  incorporating  funding  require- 
ments to  accomplish  the  effort  into  appropriate  budget  accounts.  The  process 
of  defining  the  work  is  called  programming.  Budgeting  is  the  process  of 
obtaining  approval  to  obligate  funds  to  do  the  work.  An  important  point  to 
stress  is  that  programming  and  budgeting  for  accomplishing  product  assurance 
and  test  work  is  integrated  with  the  overall  programming  and  budgeting  pro- 
cess for  the  system  acquisition.  Since  the  exercise  is  not  a separate 
entity,  it  is  important  that  a systematic  approach  be  taken  to  defining 
product  assurance  and  test  funding  requirements  for  incorporation  into  the 
overall  system  programming  and  budgeting  process.  This  means,  of  course, 
that  the  product  assurance  and  test  personnel  must  be  involved  in  the 
programming  and  budgeting  process. 

The  programming  efforts  begin  with  a definition  of  the  requirements  for 
product  assurance  and  test  work.  These  requirements  are  defined  on  a fiscal 
year  basis  for  each  work  package.  In-house  and  contractor  requirements  are 
defined  separately,  thus  giving  visibility  to  the  contractual  efforts  and 
the  in-house  management  and  support  efforts.  A product  assurance  and  test 
program  plan  and  work  breakdown  structure  are  essential  prerequisites  if 
the  estimates  of  required  funding  are  to  be  credible.  The  project  manager 
must  know  what  work  will  be  done,  when  it  will  be  done,  and  where  it  will 
be  done  in  order  to  make  decisions  regarding  the  structure  of  his  funding 


requirements  and  in  defending  the  need  for  these  requirements.  The  total 
requirements  become  a line  entry  in  the  Army's  Program  Objective  Memorandum, 
which  is  forwarded  to  Department  of  Defense  (DOD)  for  approval.  Based  on 
the  DOD  guidance,  the  Army  budget  is  prepared  and  submitted  to  DOD  for 
approval  and  transmittal  to  the  Office  of  Management  and  Budget.  Considering 
the  fact  that  it  currently  takes  about  twenty  months  from  the  time  a require- 
ment is  identified  until  budget  authority  is  received  from  Congress,  con- 
- 

siderable  forward  planning  is  required  to  anticipate  a realistic  level  of 
funding.  Also,  since  Congress  only  provides  budget  authority  for  one  fiscal 
year  at  a time,  the  programming  and  budgeting  process  is  a continuous  cycle. 
The  bottom-line  to  all  this  is  that  programming  has  a very  real  impact  on 
program  success  and  that  continued  involvement  in  programming  product 
assurance  and  test  requirements  is  essential. 

When  budget  authority  is  received  by  DOD,  a process  of  allocating  obli- 
gation authority  to  the  various  program  elements  occurs.  It  is  almost 
i he vi table  that  the  project  will  not  get  the  total  amount  of  obligation 
authority  wiich  was  originally  requested.  This  will  require  a restructuring 
of  the  program.  All  to  often,  product  assurance  and  test  functions  are  not 
fully  funded  simply  because  the  project  manager  does  not  have  visibility  of 
the  structure  and  need  for  these  requirements.  Here  again,  proper  pro- 
gramming of  the  work  can  provide  the  project  manager  with  an  insight  for 
making  appropriate  trade-off  decisions  within  the  overall  structure  of 
programmed  funding  requirements.  Figure  11  summarizes  the  programming  and 
budgeting  process  as  it  relates  to  the  product  assurance  and  test  function. 
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SECTION  VI 


MANAGEMENT  CONSIDERATIONS 


In  this  final  section  of  the  guide,  some  considerations  are  offered  on  the 
management  functions  of  planning  and  controlling  as  they  relate  to  the  pro- 
duct assurance  and  test  function. 

I Planning"!  Someone  once  said  that  any  road  will 

get  you  there  if  you  don't  know  where 

you're  going.  This  thought  aptly  sums  up  the  importance  of  planning  in  any 
endeavor.  The  key  to  successful  completion  of  any  job  is  to  set  down  the 
end  results  to  be  achieved,  lay  out  a strategy  to  get  there,  and  stick  to 
it.  All  too  often,  elaborate  plans  are  drawn  up,  only  to  function  as  a 
show-case  while  something  else,  or  nothing,  is  really  being  done.  An 
essential  feature  of  any  plan  is  that  is  must  be  structured  to  facilitate 
accomplishment  of  the  work.  Another  feature  is  that  plans  should  provide 
a basis  for  an  audit-trail  for  comparing  what  work  was  accomplished  against 
what  work  was  planned.  This  feature  is  an  essential  element  for  control 
of  the  work,  since  it  permits  the  identification  of  deviations. 

The  planning  activity  for  the  product  assurance  and  test  function  should 
be  based  on  the  work  packages.  Detailed  plans  for  each  work  package  should 
be  prepared.  These  work  package  plans  are  used  at  the  organizational  level 
where  the  work  is  performed.  The  detailed  plans  should  be  summarized  into 
an  overall  product  assurance  and  test  functional  plan.  This  plan  is  used 
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at  the  first  level  of  management  where  the  function  is  controlled.  The 
functional  plan  is  summarized  for  input  into' the  project  master  plan,  or 
overall  System  Development  Plan  for  Army  programs.  The  project  manager 
uses  this  overall  plan  to  assure  that  the  work  in  the  various  functional 
disciplines  is  being  accomplished.  This  approach  to  planning  results  in 
a hierarchy  of  plans  as  shown  in  Figure  12. 

I Control  1 Since  the  product  assurance  and  test 

function  is  based  on  the  philosophy 
of  assurance,  it  provides  the  project  manager  with  a mechanism  for  exer- 
cising control.  The  effectiveness  of  the  function  in  the  control  role  is 
enhanced  when  the  product  assurance  and  test  functional  element  is  organi- 
zationally independent  of  the  functional  elements  responsible  for  the 
technical,  cost,  and  schedule  aspects  of  the  program.  This  independence 
provides  both  Independent  assessment  and  support  roles  for  the  product 
assurance  and  test  function.  The  independent  assessment  role  can  be  used 
as  a "checks  and  balances"  system  by  the  project  manager.  Such  an  approach 
of  "checks  and  balances"  is  based  on  credible  assessments  of  the  system's 
status  and  problem  areas  throughout  the  acquisition  process.  It  is  empha- 
sized that  this  technique  provides  the  project  manager  with  a different 
perspective  of  problem  areas  and  should  be  considered  as  another  informational 
input  into  the  decision-making  process.  Periodic  assessments  are  occurring 
throughout  the  acquisition  cycle,  as  shown  in  Figure  13.  The  projtc'- 
manager  should  ask  for  a system  assessment  update  prior  to. each  major  deci- 
sion milestone,  as  a minimum. 
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I Conclusion  I 


At  the  beginning  of  this  paper,  its 
purpose  was  stated  as  one  of  developing 


a basis  for  understanding,  from  a management  viewpoint,  the  process  of 
accomplishing  product,  assurance  and  test  functions  during  Army  acquisition 
programs.  I hope  this  purpose  has  been  accomplished  within  the  scope  of 
this  paper.  Hopefully,  this  paper  will  stimulate  new  thoughts  and  ideas 
on  the  utilization  of  product  assurance  and  test  in  Army  acquisition  programs. 
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